{12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
3 January 2002 (03.01:2002) 




I 







PCT 



(10) International Publication Number 

wo 02/00011 



(51) International Patent Classification'': 



AOIJ 5/017 



(21) International Appiicatidn Number: PCT/NLOl/00471 

(22) International FUing Date: 25 June 2001 (25.06.2001) 



(25) Filing Language: 

(26) Publication Language: 

(30) Priority Data: 

1015559 



Dutch 
English 

28 June 2000 (28.06.2000) NL 



(71) Applicant ffbr all designated States except US): IDENTO 
ELECTRONICS B.V. [NL/NL]; Het Noord 6, NL-8307 
AAEns (NL). 

(72) Inventor; and 

(75) Inventor/Applicant (for US only): C6p, Jacobus, 
Petrus, Hendrikus [NL/NL]; Dobbelaan 51, NL-8521 KS 
Sl Nicolaasga (NL). 

(74) Agent: GROOTSCHOLTEN, Johannes, Antonius, 
Maria; Arnold & Siedsma, Sweelinclq>lein 1, NL-2517 
GK The Hague (NL). 



(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, OR, CU. 
CZ, DE, DK, DM, DZ, EC, EE, ES. H, GB, GD, GE, GH. 
GM, HR, HU, ID, XL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG. MK, MN, MW. 
MX, MZ, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, 
SL, TJ. TM, TR. TT. TZ. UA. UG, US, UZ. VN, YU, ZA, 

zw. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT. BE, CH, CY, DE. DK, ES. H, -FR, GB, GR, IE, 
rr, LU, MC, NL, PT, SE, TR), OAPI patent (BF, BJ, CF, 
CG, a, CM, GA, GN, GW. ML. MR. NE, SN, TD. TG). 

Published: 

— with international search report 

— before the expiration of the time limit for amending the 
claims and to be republished in the event of receipt of 
amendments 



[Continued on next page] 



(54) Title: MILKING SYSTEM WITH THREE-DIMENSIONAL IMAGING 




2^ (57) Abstract: The invention relates to a method for automatically milking livestock, comprising of determining at least the loca- 
tion of teats on an udder with imaging; airanging teat cups on the teats at the determined location with a robot arm; and milking the 
livestock with a control of the teat cups, wherein the imaging conqirises of: obtaining at least two substantially independent images 
having therein at least one common reference element; sending the images to an image-processing device; obtaining a tfaree-dimen- 
sional udder pattern from the images by means of image-processing on the basis of the reference element; and driving the robot arm 
to the teats on the basis of flie udder pattern. 
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.MIIiKlNG SYSTEM WITH .XHBEE-DIMENSXQKT^ IXMAGING 

The present invention relates to a method and 
milking system for automatically milking livestock, 
respectively comprising of determining at least the 
location of teats on an udder with imaging, arranging 
5 teat cups on the teats at the determined location with a 
robot arm; and milking the livestock with a control of 
the teat cups, and driving the robot arm toward the 
teats on the basis of formed images, and wherein the 
milking system comprises: at least two teat cups, a " 
10 robot arm for arranging the teat cups on the teats of an 
udder of a livestock animal positioned at a milking 
location, and an imaging system for determining at least 
the location of one or more than one teat from formed 
images, and a control for driving the robot arm to the 
15 teats on the basis of the formed images. 

The known method and device have the drawback that . 
they are slow. Obtaining a location to which the milking 
robot must be driven in order to arrange these teat cups 
on the teats takes much too long. The imaging techniques 
20 which are used moreover allow too many errors, which is 
a further drawback. 

The invention has for its object to partly, and 
preferably wholly, obviate the above stated drawbacks of 
the known art, for which purpose a method is provided 
25 which is distinguished by obtaining at least two 

substantially independent images having therein at least 
one common reference element, sending the images to the 
image-processing device and obtaining a three- 
dimensional udder pattern from the images for driving 
the robot arm by means of image-processing on the basis 
of the reference element; and which milking system is 
distinguished by at least one camera for obtaining at 
least two substantially independenz images having 
therein at least one common reference element, and an 
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image-processing device for obtaining a three- 
dimensional udder pattern from the images for the " 
purpose of the control by means of image-processing on 
the basis of the reference element. 

Teats of the udder are easily identifiable by 
providing a three-dimensional udder pattern. By 
subsequently deducing control information for the robot 
arm from the three-dimensional udder pattern, a rapid 
and reliable solution is provided for localizing at 
least the position of the relevant teats and preferably 
also the orientation thereof so as to enable arrangement 
of teat cups with the robot arm. 

In a preferred embodiment the method comprises of 
combining the images and herein placing the reference 
15 point in register in order to obtain the three- 
dimensional udder pattern herefrom. For this purpose the 
reference point can be made identifiable in the 
different images in diverse ways. A light source can 
thus be used to form a brightly illuminated dot on the 
relevant udder which can then be identified in both 
images of the udder. In this manner it is possible to 
place the reference points in register in the images. 
Use can be made according to the method of one camera, 
which is then displaced between obtaining the 
substantially independent images and redirected to the 
presumed location of the udder, or two cameras can be 
used. 

In another preferred embodiment the method 
comprises of placing the camera at a distance and 
carrying an image for recording to the camera using a 
light conductor. Irrespectively of whether one. or two 
cameras are used, the object here is that the images are 
as independent as possible. When the images are obtained 
from positions at angles of view which enclose a spatial 
35 angle of 90°, as much information as possible about the 
udder pattern is obtained with the images. The udder 
pattern is then for instance a relief map or contour map 
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■in which the teats are clearly identifiable. Such a 
xelief image is preferably generated from independent 
-images with an overlap of for instance 60-80%. 

The camera (s) is or are preferably disposed at a 
5 distance from the milking location. At the desired 
positions from which images of the udder must be 
obtained under the above stated conditions and preferred 
circumstances, it is then only necessary for instance to 
arrange lenses wherein light conductors extend between 
10 the lenses and the actual cameras." The cameras can be 
CCDs. Thus is ensured that a restless animal cannot 
damage the cameras, but only the simple lenses, which' 
can be replaced at low cost if the restless animal steps 
against them. 

The invention will be further elucidated 
hereinbelow on the basis of a number of embodiments as 
shown in the drawing, in which: 

fig. 1 and 3 show a perspective view of a milking 
system according to the present invention, in which the 
method is also implemented; 

fig. 2 shows a perspective view of an alternative 
device; and 

fig. 4 shows a second alternative embodiment 
relative to fig. 2. 

Fig. 1 shows a part of a milking parlour in which 
an industrial robot 1 is disposed. Robot 1 is situated 
for travel on rails 2 in a milking pit 3, through which 
the farmer traditionally walked in order to arrange the 
teat cups, this now being done with industrial robot 1. 

As already noted, robot 1 arranges teat cups 4 on 
teats 5 of udder 6 of a cow 7 standing in a milking box 
8. Robot 1 can travel in order to arrange teat cups 4 in 
diverse milking boxes 8. 

As described in more detail hereinbelow and in fig. 
2-4, robot 1 is provided with at least one camera 9; in 
the case shown in fig. i two cameras are arranged. The 
cameras can for instance be of the CCD type, although 
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other imaging devices are also pos&ibie, insofar the 
signals originating therefrom can be processed by a 
control 10 as shown schematically in fig. 2. 

As shown in fig, 2, two images of the udder are 
5 obtained substantially simultaneously at angles of view 

a 

enclosing a spatial angle a. In order to enable the 
collection of as much information as possible about the 
shape of the udder, the position of the teats, the 
orientation of the teats and so on, the obtained images 
10 must be as independent as possible. For this reason the 
spatial angle a enclosed between the directions of view 
amounts preferably to at least approximately 90**. 

Further shown in fig. 2 is that only lenses 11 are 
arranged on arm 12 of the robot, and that cameras 9 are 
15 placed at a distance. Light conductors 13 are arranged 
between lenses 11 and the actual cameras 9. Damage to 
the relatively expensive cameras 9 by a restless animal 
for milking is effectively prevented with such a 
configuration. Light conductors 13 can for instance be 
20 glass fibre cables. 

Fig. 2 further shows that a light source 14 is 
arranged on arm 12. Light source 14 illuminates a 
reference point 15 which is identifiable in both images 
of cameras 9- With this reference point 15 in both 
25 images of cameras 9 a three-dimensional udder pattern 
can be deduced using an image-processing program in 
control 10. The three-dimensional udder pattern is 
generated by the control in the form of for instance a 
relief map or contour map. It is noted in this respect 
30 that the arrangements shown in the figures are 

schematic, since it must be possible to map the whole 
udder with one set of images. The cameras 9 (or lenses 
11) used are placed, at a distance from the presumed 
location of. udder 6 such that such an imaging is 
35 possible. A stereoscopic image of udder 6, and more 

particularly the teats 5 thereof, is thus generated with 
the two cameras {or lenses) . 
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As will be apparent from the foregoing, control 10 
comprises an image-processing device for obtaining the 

pattern. Control 10 also 
provides the driving of .robot 1 for the purpose of 
5 arranging teat cups 4 on the teats. For the purpose of a 
clear view in the drawing, the robot 1 does not carry a 
teat cup 4. It is on the other hand advantageous to have 
a teat cup at the ready at the time of the imaging. In 
this way little time need elapse between forming of the 
10 images and the arranging of teat cups 4 on teats 5, so 
that there remains only a limited chance of the cow 
moving, which will necessitate a hew imaging. 

Fig. 3 shows the embodiment of fig. i in more 
detail, wherein cameras 9 are arranged directly on robot 
arm 12-. Cameras 9 can herein be directed, this being 
indicated with double arrows A. An extendable part 16 of 
robot arm 12 is further provided, with which the 
distance between cameras 9 can be varied. In the 
embodiment shown here an optimally flexible 
configuration is obtained by variation of the distance" 
between cameras 9 with part 16 and the adjustment of the 
associated angles of view along arrows A. Adjustment of 
cameras 9 and extension of part 16 of robot arm 12 are 
carried out by control 10 (not shown in fig. 3) . A 
configuration similar to that of fig. 2 is of course 
possible, wherein cameras 9 are not placed on the robot 
arm, but at a distance. 

Fig. 4 shows an alternative embodiment with a 
single camera 9 on the extendable part 16 of robot arm 
12. The extendable part is displaceable in the direction 
of arrow B. The single camera is again adjustable, i.e. 
the angle of view thereof is adjustable. In a retracted 
position of the extendable part 16 of robot arm 12 the 
camera 9 is directed forward relative to robot arm 12, 
as shown here with dash-dot lines. Conversely, camera 9 
IS directed to the rear relative to robot arm 12 in the 
extended position of extendable part 16. Two successive 
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^- - « ^v^-^---»'^ a v»T-v short time with a 
images can tnus ne ODL-axixewx u.** a a, — y 

single camera 9, with only a very limited risk of the 
relevant animal moving in the meantime . The embodiment 
of fig. 4 is different from the foregoing embodiments in 
5 that only a single camera 9 is provided, which must 

obtain two successive images . Although between obtaining 
of the images some time elapses during which an animal 
for milking could move, only a single camera 9 is 
required in this embodiment of fig. 4, which represents 
10 a saving relative to the foregoing embodiments. 

The extendable part 16 of robot arm 12 thus forms 
displacing means for displacing the camera, as well as a 
carrier for placing the single camera. 

In the embodiments of fig. 3 and 4 the reference 
point 15 is furthermore not an explicitly illuminated 
point, but another clearly identifiable element in both 
images. A discolouration of the skin, an abrupt bend in • 
a blood vessel lying close to the surface of the skin, 
and so on can be envisaged here. In the embodiment of 
fig. 3 and 4 no separate light source is therefore 
provided for the reference point, although a' separate 
light source (not shown) can still be provided to 

illuminate the whole udder. 

Many alternative and additional embodiments are 
25 possible within the scope of the present invention as 
defined in the appended claims. Both or all cameras can 
be in stationary disposition relative to each other m 
• an embodiment with two or more cameras . More than two 
images can also be used, and more than one reference 
30 point can be used in the images to combine the images 
into the desired three-dimensional udder pattern so as 
to enable deducing therefrom of the position and 
optionally also the. orientation of the different teats, 
in addition, cameras can be used ouher than the stated 
35 cameras of the CCD-type. If more than two cameras are 
used; it is still recommended to keep the directions of 
view as independent as possible. 
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In the -foregoing an industrial robot is used by way 
of example in each case. Other milking robots of a more 
conventional type can also be used to apply the present 
invention. The invention can further be applied with 
animals other than cows, such as goats and sheep and so 
on. Displacing means other than an extendable part of 
the robot arm can also be used in the embodiments of 
fig. 3 and 4. Displacement of the camera over the robot 
arm can for instance be envisaged along for instance a 
guide rail or a similar configuration. It is thus 
apparent that many alternative and additional 
embodiments will occur to the skilled person after • 
examination of the foregoing. 

The embodiments of the invention explicitly 
described above and shown in the drawings are therefore 
not definitive, but the invention is limited solely to 
the definition according to the appended claims. 
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1. Method "for automatically milking livestock, 

m 

comprising of determining at least the location of teats 
on an udder with imaging; arranging teat cups on the 
teats at the determined location with a robot arm; and 
5 milking the livestock with a control of the teat cups, 
wherein the imaging comprises of: obtaining at least two 
siibstantially independent images having therein at least ^ 
one common reference element; sending the images to an ) 
image-processing device; obtaining a three-dimensional 
10 udder pattern from the iiaages by means of image- 
processing on the basis of the reference element; and 

driving the robot arm to the teats on the basis of the !: j 

udder pattern. \: 

2. Method as claimed in claim 1, comprising of j 
15 combining the images and herein placing the reference 1: \ 

point in register in order to obtain the three- ; : 

dimensional udder pattern herefrom. 

3. Method as claimed in claim 1 or 2, wherein 

obtaining the images comprises of: activating at least ;.j 
20 one camera. 

4- Method as claimed in claim 3, comprising of 
activating substantially simultaneously two cameras 
directed at an angle known beforehand relative to each 
other . 

25 5. Method as claimed in claim 4, comprising of 

activating one camera at least twice in succession. 

6- Method as claimed in claim 5, comprising of 
displacing the camera between obtaining the images. 
7. Method as claimed in claim 5 or 6, further 
30 comprising of displacing the camera over a known 
distance and, prior to recording of an image, 
. redirecting the camera to the presumed location of the 
udder in order to obtain at least two images at an angle 
known beforehand. • 
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8 . Method as claimed in any of the foregoing claims 
3-6, comprising of placing the camera at a distance and 
carrying an image for recording to the camera using a 
light conductor . 

9. Milking system for implementing a method as 
claimed in one or more than one of the foregoing claims 
for substantially automatic milking of livestock, 
comprising: at least two teat cups; a robot arm for 
arranging the teat cups on teats of an udder of a 
livestock animal positioned at a milking location; and 
an imaging system which comprises: at least one camera 
for obtaining at least two substantially independent- 
images having therein at least one common reference 
element; an image-processing device for obtaining a 
three-dimensional udder pattern from the images by means 
of image-processing on the basis of the reference 
element; and a control for driving the robot arm to the 
teats on the basis of the three-dimensional udder 
pattern. 

10. Milking system as claimed in claim 9, wherein 
the imaging system comprises at least two cameras which 
are arranged directed at an angle known beforehand. 

11. Milking system as claimed in claim 10, wherein 
the angle amounts to at least approximately 90°. 

12. Milking system as claimed in claim 9 with one 
camera, wherein the imaging system comprises a carrier 
for the camera, which carrier comprises displacing means 
for displacing the camera in the period between 
recording successive images of the udder. 

13. Milking system as claimed in claim 12, wherein 
the carrier or the camera comprises directing means with 
which the camera can be directed to a location known 
beforehand of the udder of a relevant animal. 

■ 14. Milking system as claimed in one or more than 
one of the foregoing claims 9-13, wherein the camera is 
disposed at a distance from the milking location. 
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15. Milking system as claimed in claim 14, wherein 
a light conductor directed or to be directed at the 
location of the udder is arranged in the vicinity of a 
presumed location of the udder of an animal for milking, 

5 which light conductor extends to the camera at a 
distance from the milking location. 

16. Milking system as claimed in claim 15, wherein 
an imaging element, such as a lens, is arranged on the 
end of the light conductor directed or to be directed 

.10 toward the udder at the presumed location. 
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